User choice as an evaluation metric for web translation services in cross language instant messaging applications

A method for evaluating MT performance embedded in Cross-Language Instant Messaging (CLIM) systems is presented. A web interface that provided concurrent realtime translation for instant messaging from multiple MT services was developed and used by paid participants to collaborate on a photo identification task. The method showed a task performance benefit due to the availability of multiple translation alternatives. The method also provides a new evaluation metric for MT systems based on user‟s task motivated choices. This method was used to compare two English-Japanese online translation systems, one from Google, and one from Excite/Japan.